Back to RGB: 3D tracking of hands and hand-object interactions based on short-baseline stereo

摘要

We present a novel solution to the problem of 3D tracking of the articulatedmotion of human hand(s), possibly in interaction with other objects. The vastmajority of contemporary relevant work capitalizes on depth informationprovided by RGBD cameras. In this work, we show that accurate and efficient 3Dhand tracking is possible, even for the case of RGB stereo. A straightforwardapproach for solving the problem based on such input would be to first recoverdepth and then apply a state of the art depth-based 3D hand tracking method.Unfortunately, this does not work well in practice because the stereo-based,dense 3D reconstruction of hands is far less accurate than the one obtained byRGBD cameras. Our approach bypasses 3D reconstruction and follows a completelydifferent route: 3D hand tracking is formulated as an optimization problemwhose solution is the hand configuration that maximizes the color consistencybetween the two views of the hand. We demonstrate the applicability of ourmethod for real time tracking of a single hand, of a hand manipulating anobject and of two interacting hands. The method has been evaluatedquantitatively on standard datasets and in comparison to relevant, state of theart RGBD-based approaches. The obtained results demonstrate that the proposedstereo-based method performs equally well to its RGBD-based competitors, and insome cases, it even outperforms them.